1. Field of the Invention
The present invention relates to techniques for estimating a position and orientation of a camera that picks an image of a real space or an object movable in the real space.
2. Description of the Related Art
The measurement of a position and orientation of the image-pickup section of a camera that picks an image of a real space (hereinafter, the image-pickup section of a camera is simply referred to as a “camera”), is needed for a mixed reality system that displays a real space and a virtual space in an integral manner. A position and position sensor, such as a magnetic sensor, is used for the measurement of the position and orientation of a camera. Its measurement accuracy, however, is not sufficient for being used in the mixed reality system. With this being the situation, the improvement in its measurement accuracy with respect to position and orientation is usually effected by correcting measurement values. As such correction methods, the following four types of methods have been proposed.
First Method
A position and orientation of a camera are measured by a position and orientation sensor such as a magnetic sensor mounted on a camera, and the measurement error of the position and orientation sensor is corrected by using markers of which the positions in a real space are known, or characteristic points of which the positions disposed in a real space are known (hereinafter, markers and characteristic points are collectively referred to as “indices”). This correction method is disclosed in the following patent documents: (1) Japanese Patent Laid-Open No. 11-136706 (corresponding U.S. Pat. No. 6,522,312) (hereinafter “JP 11-136706”), (2) Japanese Patent Laid-Open No. 2000-041173, and (3) Japanese Patent Laid-Open No. 2002-228442 (corresponding U.S. Published patent application No. US20020103617 A1). The present correction method is also set forth in the following non-patent documents: (1) Bajura Michael and Ulrich Neuman: “Dynamic registration correction in video-based-augmented reality systems”, IEEE Computer Graphics and Applications 15,5, pp. 52-60, 1995; and (2) A. State, G. Hirota, D. T. Chen, B. Garrett, and M. Livingston: “Superior augmented reality registration by integrating landmark tracking and magnetic tracking”, Proc. SIGGRAPH '96, pp. 429-438, July 1996 (hereinafter, “A. State”). This correction method, perceiving it as a problem that the position and orientation sensor, such as a magnetic sensor, is lacking in the measurement accuracy with respect to the position and orientation of a camera used in a mixed reality system, corrects measured values by the sensor using information about indices themselves and information based on the detected results of indices from an image. That is, the first method is a method for correcting measured values by the position and orientation sensor using indices, and is effective as a method for performing high-accuracy position and orientation measurement.
Second Method
In the second method, when a measurement error of the sensor is to be corrected, the correction values for a current frame are calculated by using the correction values for a preceding frame and those for the current frame. This method is disclosed in the aforementioned patent document (1) (JP 11-136706). The second method is used when a plurality of indices is employed, and serves as a processing method for changing over between employed indices. According to this method, the discontinuity of correction values in making a changeover between indices can be smoothed. Thus, in the second method, the correction value for the current frame is calculated by making use of the correction values for the preceding frame and those for the current frame.
Third Method
The third method is a method proposed by Japanese Patent Application No. 2002-081251. According to circumstances, the estimating method for the position and orientation of a camera is changed over. Possible methods for determining a position and orientation include a variety of methods, each having its own advantage and disadvantage. The third method aims to perform a better position and orientation estimation by using a method obtained by selectively combining these “methods for determining a position and orientation”, that is, by changing over the position and orientation estimating method as circumstances demand.
Fourth Method
The fourth method is a method for bridging the gap in measured value between a state where a correction by indices is provided and a state where a correction by indices is not provided (i.e., a state where measured values of a position and orientation sensor alone are provided). This method is set forth in the second paragraph in Section 7.1 in the aforementioned non-patent document (2) (A. State). According to a brief explanation of this document, when the situation is changed from a state where a correction by an image is provided to a state where a correction by an image is not provided, the estimation results of the position and orientation of a camera are gradually shifted to the measured values by the sensor. According to the method set forth in this non-patent document (2) (A. State), when the detection of indices eventuates in failure, the estimation results of the position and orientation are to be shifted to sensor's measurement values themselves having large errors.
Possible concrete techniques of the first method include a variety of techniques, each having its own advantage and disadvantage depending on conditions of indices captured by a camera. Accordingly, as in the third method, it is desirable to change over the correction method as circumstances demand. The third method is not limited to the techniques of the first method that corrects measurement values by the position and orientation sensor using indices, but it is a method applicable to the entire spectrum of the estimation of position and orientation. The third method would also be applicable to a case where techniques that are different from each other in correction method of the first method are mutually changed over therebetween. However, when the third method is applied, just as it is, to a method that is different in correction method, correction results of a position and orientation undesirably result in discontinuous values the moment when the correction method is changed over therebetween. This leaves room for improvement.
On the other hand, as a method that allows for position and orientation estimated values to become discontinuous, the second method 2 is proposed by the correction method disclosed in the patent document (1) (JP 11-136706). However, the second method is an invention that, when using a plurality of indices, perceives, as a problem, the discontinuity of position and orientation estimated values occurring when indices are changed over, and that solves this discontinuity. The second method does not go so far as to allow for and solve the changeover of the correction method of position and orientation.
The fourth method seems like processing without any problem. However, when, for example, an index is in a state of being sufficiently captured by a camera, if it is assumed that part of the index is hidden by hand without moving the camera, it will become apparent that this method has a problem. More specifically, in such a case, the estimation results should be originally the same between before and after the index is hidden by hand. However, according to the technique of the fourth method, if the index becomes non-capturable, sensor's measurement values are misidentified as position and orientation estimated values. As a result, although high accuracy position and orientation estimation has been secured before hiding the indices, the hiding of the index unfavorably brings about low accuracy estimation results including the errors inherent in sensor's measurement values in their entirety.